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1.  INTRODUCTION 


A.  Objective 

The  objective  of  creating  a  computer  simulated  virtual  environment  for 
medical  training  is  to  provide  a  level  of  training  not  possible  using  traditional 
methods.  In  this  project,  we  are  developing  the  software  tools  and  the 
medical  content  to  produce  a  virtual  reality  simulation  of  the  surgical 
removal  of  the  "Shattered  Kidney",  complete  with  all  of  the  visual  and  haptic 
features  of  the  actual  surgical  procedure.  This  procedure  is  a  critical 
component  of  trauma  management  in  the  battlefield  setting,  where  massive 
blunt  or  multiple  penetrating  injury  requires  the  surgical  removal  of  part  or 
all  of  the  kidney  (nephrectomy).  In  addition  to  the  procedure  itself,  special 
tracking  protocols  will  be  embedded  in  the  application,  so  that  the 
performance  of  individual  users  can  be  evaluated,  and  compared  to  a  database 
of  normative  values. 

B.  Background 

Traditional  techniques  for  teaching  trauma  procedures  have  depended  largely 
on  the  existence  of  a  sufficiently  large  number  of  proctors  with  adequate 
surgical  skills  to  teach  trauma  procedures.  Other  approaches  include  practice 
on  animals,  but  animal  models  of  injury  often  do  not  reflect  human  trauma, 
and  raise  a  host  of  ethical  issues  concerning  procuring  and  maintaining 
animals  for  surgical  training.  Also,  practice  on  humans  and  animals 
precludes  the  ability  to  repeatedly  rehearse  specific  components  of  the 
procedure  that  may  prove  challenging  or  require  finely  tuned  motor  skills. 
An  additional  concern  is  that  Department  of  Defense  (DoD)  hospitals  are 
usually  not  regional  trauma  centers,  so  that  physicians  and  allied  health 
personnel  in  the  military  may  not  obtain  significant  exposure  to  human 
trauma  cases  for  training  purposes. 

Although  the  airline  industry  and  DoD  have  used  flight  and  battlefield^ 
simulators  for  many  years,  several  technical  challenges  have  limited  the  use 
of  computer-based  simulation  technology  in  medical  education.  A  flight 
simulator  is  easier  to  implement  than  a  surgical  simulator;  the  terrain  of  the 
ground  is  fixed  and  rigid,  and  an  airplane  simply  moves  through  a  path 
above  this  terrain.  A  surgical  simulator,  on  the  other  hand,  is  more  complex. 
The  terrain  of  the  body  (the  internal  organs)  must  be  interacted  with,  and  they 
must  flex,  be  able  to  be  cut  and  then  re-attached.  This  manipulation  involves 
much  greater  computational  sophistication.  The  organs  in  the  body  must  be 
programmed  with  physiological  behaviors  and  basic  principles  of  physics  so 
they  respond  appropriately  when  they  are  cut,  tugged,  and  stretched.  In 
addition,  the  surgical  simulator  must  have  knowledge  of  how  each 
instrument  interacts  with  the  tissues.  For  example,  a  scalpel  will  cut  tissue 
when  a  certain  amount  of  pressure  is  applied;  however,  a  blunt  instrument 
may  not-  this  fact  must  be  simulated. 
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Virtual  reality  technology  holds  tremendous  promise  for  surgical  trauma 
training,  because  it  offers  physicians,  battlefield  and  emergency  medical 
personnel  the  opportunity  to  practice  in  an  environment  where  mistakes  do 
not  adversely  affect  patients.  An  optimal  training  simulator  accurately 
replicates  the  physical  and  physiological  properties  of  the  real  procedure.  In 
addition,  it  offers  the  ability  to  automatically  track  and  evaluate  performance, 
provides  the  option  of  different  procedural  scenarios,  as  well  as  simulating  a 
range  of  surgical  complications  and  anatomical  anomalies. 

Computerized  surgical  simulations  will  make  a  tremendous  impact  in 
improving  surgical  morbidity  and  mortality.  Studies  have  shown  that,  for  a 
wide  range  of  diagnostic  and  therapeutic  procedures,  doctors  doing  their  first 
few  to  several  dozen  cases  are  much  more  likely  to  make  a  greater  number  of 
errors.  Adequate  proctoring  of  learners  by  experienced  surgeons  is 
cumbersome,  as  there  are  few  surgeons  experienced  enough  in  the  techniques 
to  proctor  their  colleagues.  It  is  exceedingly  difficult,,  for  physicians, 
particularly  those  in  rural  areas,  to  travel  to  larger  medical  centers  for 
training.  The  requirement  also  places  a  burden  on  experts  who  could  become 
overwhelmed  with  proctoring  requests. 

C.  Technical  Goals  and  Experimental  Approaches 

The  goal  of  this  project  is  to  make  the  "Shattered  Kidney"  simulator  as 
realistic  as  possible  for  a  positive  training  experience.  Realism  in  the  context 
of  virtual  environments  for  medical  training  relates  to  how  the  anatomical 
structures  appear,  how  life-like  the  interaction  is  with  the  anatomy,  and  how 
it  behaves  when  one  interacts  with  the  anatomy.  To  accomplish  this  goal,  the 
first  requirement  is  the  development  of  software  that  can  produce 
anatomically  accurate  models  that  display  the  physical  and  physiological 
characteristics  of  living  tissues. 

The  first  technical  goal  of  the  project  is  to  develop  a  software  authoring 
environment  in  which  physical  attributes  can  be  integrated  into  the 
anatomical  models,  virtual  surgical  tools  and  input  device,  so  that  all  behave 
in  an  appropriate  manner.  This  involves  examination  of  the  surgical 
procedure  to  identify  those  actions  that  require  enhancement  of  the  current 
technology,  such  as  cutting,  clamping,  grasping  and  suturing,  as  well  as 
specification  of  the  normal  behavior  of  the  tissue.  New  algorithms  and 
physical  solutions  are  needed  to  simulate  all  of  these  actions.  The  general 
approach  is  based  on  advances  in  'physically'  or  'physics-based'  modeling,  as 
articulated  by  Barr  and  his  colleagues  (see  Barzel,  1992). 

The  abdominal  virtual  environment  is  created  from  the  magnetic  resonance 
imaging  (MRl),  computerized  tomography  (CT),  and  the  photographic  dataset 
of  the  Visible  Human  Project  (National  Library  of  Medicine).  The  surgery 
scene  consists  of  several  types  of  entities  including  physically-based  tubes. 
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polyhedra,  volumetric  data,  and  particle  systems.  Tubes  are  the  primary  entity 
of  a  simulation  and  embody  a  spatial  spline  formulation  having  the 
properties  of  Newtonian  physics.  They  have  the  ability  to  be  deformed  in  real¬ 
time  through  contact  with  tools,  other  tubes,  and  additional  forces.  Tubes  can 
be  developed  from  the  medical  imagery  templates  as  well  as  independently 
created.  Polyhedra  are  non-deformable  objects  that  may  te  imported  from 
standard  file  formats  and  manipulated  to  be  properly  scaled  and  placed  into 
the  surgical  scene.  Volumetric  data  is  derived  from  the  same  imagery  used  for 
templates  and  can  be  used  to  view  the  interior  of  objects  by  taking  advantage 
of  the  3D  volume  texture  rendering  capabilities  of  Silicon  Graphics 
workstations. 

Once  the  geometry  of  a  surgical  scene  has  been  defined,  the  interconnections 
between  the  objects  and  the  physical  properties  of  the  tubes  are  specified. 
Surgical  tools  are  selected  from  a  group  of  previously  created  tools  or  a  new 
tool  can  be  created.  Lastly,  the  simulation  designer  refines  the  response  of  the 
simulation  by  testing  and  adjusting  the  interconnections  and  their  properties. 
Once  a  simulation  is  complete  all  that  needs  to  be  provided  to  the  trainee  is 
the  data  describing  the  scene  and  the  simulation  engine.  The  simulation 
engine  allows  a  simulation  to  be  paused  or  saved  for  later  use.  The  activities 
of  the  simulation  user  can  also  be  saved  for  later  review.  An  additional 
feature  of  the  simulation  engine  is  the  ability  to  save  'snapshots'  for  the 
simulation  in  progress  that  can  be  viewed  with  the  recently  developed  3D 
Internet  web  browsers. 

The  second  technical  goal  of  the  project  is  to  add  functionality  to  the  software 
authoring  environment  so  that  physiological  characteristics  of  the  tissue  can 
be  modeled.  Once  the  3D  models  have  been  created  from  2D  image  sources, 
one  of  the  biggest  challenges  for  surgical  simulation  is  creating  organs  that 
exhibit  the  physiological  characteristics  of  living  tissues.  These  properties 
include  intrinsic,  ongoing  autonomic  functions  such  as  peristalsis  of  the 
gastrointestinal  tract,  and  physiologic  responses  to  extrinsic  perturbation  of 
tissues  by  cutting,  grasping  and  pulling  with  surgical  instruments. 

Among  the  feature  set  under  development  in  the  software  authoring  toolkit, 
we  have  included  the  following  physiological  characteristics: 

(1)  Localized  Irritability  -  This  is  a  general  property  of  most  organs,  and 
consists  of  a  localized  contraction  of  the  surface  ('twitch')  in  response  to  being 
touched  or  prodded.  This  response  is  greater  in  reaction  to  metal  instruments, 
because  the  response  is  electrophysiologic  in  nature  (mediated  by  local 
neurons  or  muscle  cells).  Other  localized  responses  include  bleeding 
following  cutting  or  damage  caused  by  other  tissue  manipulations. 
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(2)  Reflexivity  - 

This  covers  a  magnitude  of  responses  that  are  mediated  by  reflex  circuits  that 
travel  through  the  central  nervous  system.  One  example  is  the  contraction  of 
a  muscle  after  being  grasped  and  pulled.  Almost  all  tissue  responds  to  painful 
stimuli  with  a  variety  of  actions,  such  as  withdrawal  from  the  stimulus,  and 
may  exhibit  more  global  responses,  which  at  a  local  organ  level  would  be 
manifested  by  an  increase  in  pulse  rate  and  respiratory  movement. 

(3)  Contractility  - 

This  is  a  feature  of  muscle  (cardiac,  smooth  and  skeletal).  One  obvious 
example  is  the  rhythmic  contraction  of  the  heart,  which  produces  the 
pulsatile  pressure  wave  in  blood  vessels  (the  magnitude  of  the  pressure  wave 
is  greater  in  arteries  than  in  veins). 

(4)  Respiration  - 

A  property  of  the  respiratory  system,  which  can  be  rapidly  modified  by 
external  stimuli  such  as  exercise  and  painful  stimuli.  Respiration,  like  heart 
rate,  is  an  ongoing  phenomenon  which  is  obvious  at  the  local  organ  level. 

(5)  Peristalsis 

This  list  is  not  exhaustive,  and  other  local  and  global  physiological  properties 
are  being  added  to  the  repertoire  of  templates  and  functions  during  the 
ongoing  evolution  of  the  software  authoring  toolkit. 

The  third  technical  goal  of  the  project  is  the  integration  of  all  physical  and 
physiological  functionality  with  the  medical  content  of  the  "Shattered 
Kidney"  procedure  to  produce  the  virtual  reality  simulation.  Once  the 
software  features  of  the  authoring  toolkit  have  been  developed,  a  full 
treatment  of  the  "Shattered  Kidney"  procedure  will  be  authored,  in  concert 
with  medical  consultants  such  as  Dr.  Howard  R.  Champion,  Director  of 
Trauma  Surgery  at  the  Uniformed  Health  Sciences  University.  Storyboards 
will  be  prepared  from  the  procedural  script,  and  all  of  the  virtual  reality  and 
interactive  multimedia  elements  will  be  implemented  based  on  this  design. 
Hypermedia  elements  will  include  audio  cues  such  as  'Warnings'  when 
mistakes  are  made,  and  text  screens  providing  case  presentations  and 
reference  information.  Performance  tracking  software  will  be  embedded  into 
the  procedure  to  measure  timing  and  accuracy  of  individual  users.  The 
performance  assessment  will  be  configured  to  provide  both  individual 
feedback  to  the  trainee,  as  well  as  administration  by  a  mentor  or  supervisor. 
Performance  media  elements  will  include  registration,  administration  and 
outcome  analysis  screens. 
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ILBODY 


A.  Technology  Development  for  Trauma  Simulation 

Much  of  the  first  year's  efforts  have  been  directed  towards  building 
programming  tools  that  will  allow  us  to  produce  a  completely  realistic, 
interactive  abdominal  simulation  of  the  "Shattered  Kidney"  procedure.  The 
rapid  pace  of  technological  innovation  in  graphics  and  physically-based 
modeling  has  led  us  to  concentrate  on  the  creation  and  optimization  of 
programming  tools  and  libraries,  and  has  prompted  us  to  devote  most  of  our 
efforts  in  this  arena  prior  to  the  full  implementation  of  the  "Shattered 
Kidney"  simulator.  A  software  authoring  toolkit  is  being  developed  that 
allows  both  the  programmer  and  the  non-programmer  to  produce 
interactive,  physically-based  simulations  using  pre-determined  or  customized 
medical  content.  Models  can  be  built  from  scratch,  or  derived  from  sequential 
2D  imagery  such  as  CT,  MR  and  the  'Visible  Human'  dataset.'^The  architecture 
consists  of  simulation  design  and  simulation  engines  for  the  creation  and 
manipulation  of  virtual  environments.  The  software  uses  OpenGL  and 
Openinventor,  and  is  configured  to  run  on  Silicon  Graphics,  Inc.  (SGI) 
workstations.  The  interface  is  designed  to  accept  'force-feedback'  and  passive 
input  devices  for  a  wide  variety  of  procedures,  ranging  from  endoscopy  to 
minimally  invasive  surgery. 

(1)  Progress  Report 

We  have  made  progress  on  the  following  tasks  within  the  first  year  of  the 
project  period: 

Planning  Tasks 

•  Assess  open  kidney  repair  surgical  procedure  for  exact  actions 

•  Survey  of  existing  simulation  technologies 

•  Assessment  of  software  enhancements  needed  to  perform  actions 

•  Evaluation  of  algorithmic  and  coding  solutions  for  software 
enhancements 

•  Identification  of  subsections  of  full  surgical  procedure 
Code  Implementation  Tasks 

•  Rearchitecting  software  authoring  toolkit 

•  Surgery  simulation  authoring  software  tools 

•  Surgery  simulation  interaction  application 

•  Tools  to  create  anatomic  geometry  from  Visible  Human  data  set 

•  Tools  to  create  realistic  textures  for  anatomy 

•  Integration  of  force  feedback  equipment  with  the  simulation  code 
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Surgery  Content  Development  Task 


•  Creation  of  normal  anatomy  for  open  kidney  procedure  simulation 

Our  primary  accomplishments  this  past  year  related  to  the  development  of 
software  tools  for  the  ''Shattered  Kidney"  procedural  simulator  have 
included: 

Planning  Tasks: 

•  Assessment  of  the  "Shattered  Kidney"  surgical  procedure  for  exact  actions 

A  full  literature  survey  coupled  with  consultation  with  trauma  surgeons 
have  been  undertaken  to  identify  the  set  of  specific  actions  required  to 
perform  the  "Shattered  Kidney"  procedure.  The  deliverables  for  this 
component,  due  by  February  1996,  include: 

•  Procedure  description  and  script 

•  List  of  physical  tasks  to  be  simulated 

•  List  of  anatomical  structures  involved  in  procedure 

•  Survey  of  existing  simulation  technologies 

A  broad  market  survey  of  available  simulation  development  software  is 
underway.  New  technologies  are  being  evaluated  to  determine  any  existing 
tools  that  can  be  used  to  help  create  the  "Shattered  Kidney"  procedural 
simulation.  Spatial  tracking  and  force  feedback  devices  have  been  evaluated 
to  find  most  applicable  interface  devices  for  simulation.  Computer  hardware 
developments  including  CPU's,  dedicated  graphics  display  boards,  and 
operating  systems  have  been  evaluated  as  part  of  this  process.  This  survey 
will  remain  as  a  continuing  effort  throughout  the  project  to  assure  that  the 
most  effective  technologies  are  being  used.  The  deliverables  include  update 
reports  internally  of  software  and  hardware  technologies 

•  Assessment  of  software  enhancements  needed  to  perform  actions 

After  assessment  of  the  physical  tasks  associated  with  the  "Shattered  Kidney" 
procedure,  we  have  identified  a  set  of  actions  that  cannot  be  satisfactorily 
simulated  by  the  current  software  algorithms.  These  actions  have  been 
broken  down  into  specific  features  that  must  belong  to  the  different  objects 
simulated  within  the  software  authoring  environment. 

New  actions  to  be  simulated  include: 

•  unstructed  cutting  of  deformable  tube  objects 

•  bleeding  -  fluid  modeling  using  particle  systems 

•  fine  and  blunt  dissection 

•  suturing 

•  ligation 
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•  vascular  clamping 

•  modeling  of  physiological  events,  including  arterial  pulsation  and 
intestinal  peristalsis 

•  Evaluation  of  algorithmic  and  coding  solutions  for  software  enhancements 

As  new  features  are  to  implemented,  the  applicable  literature  has  been 
surveyed  to  evaluate  previous  related  algorithmic  approaches.  Either 
previous  algorithms  have  been  adopted  or  new  solutions  have  been  derived 
as  needed.  In  each  case  the  feature  has  been  converted  into  implemented 
code.  The  deliverable  for  this  evaluation  component  will  be  a  report  to  be 
generated  by  the  end  of  October  1996. 

•  Identification  of  subsections  of  full  surgical  procedure 

The  "Shattered  Kidney"  procedure  has  been  broken  down  into  smaller 
subsections.  These  subsections  are  being  prioritized  based  oh  the  complexity 
of  the  simulation  task  in  relation  to  the  existing  capabilities  of  the  software 
authoring  environment.  The  purpose  is  to  produce  simulations  of  parts  of 
the  procedure  that  are  easier  to  simulate  early  on  in  the  project  so  that  there  is 
a  series  of  demonstrable  results  throughout  the  project. 

The  major  subsections  include: 

•  cutting  open  the  abdominal  wall,  retraction 

•  abdominal  exploration 

•  fine  dissection  of  major  vessels  and  tissue  planes 

•  vascular  clamping 

•  colon  resection 

•  kidney  repair 

Code  Implementation  Tasks: 

•  Rearchitecting  software  authoring  toolkit 

The  current  architecture  of  the  algorithms  and  code  in  the  software  authoring 
toolkit  has  been  assessed  and  we  have  determined  what  must  be  changed  and 
how  this  must  occur.  The  total  rewrite  of  the  existing  simulation  code  is 
driven  by  the  need  to  make  the  code  upon  which  the  open  kidney  procedure 
simulation  to  be: 

•  Robust-  will  not  fail  during  simulation,  will  always  behave  in  an 
expected  manner 

•  Generalized-  capable  of  simulating  other  surgeries  more  easily  in  the 
future 

•  Open  ended-  no  impediments  to  future  enhancements  to  the 
simulation  abilities 

•  Platform  independent-  capable  of  being  moved  to  more  cost  effective 
hardware 
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The  deliverable  for  this  component  will  be  the  new  version  of  core  code 
implementing  all  the  above  principles. 

•  Surgery  simulation  authoring  software  tools 

Taking  the  core  simulation  code,  an  application  is  being  created  that  allows  a 
non-programming  user  who  is  or  has  access  to  the  medical  content  experts  to 
create  their  own  surgery  simulation.  This  will  enable  HT  as  well  as  future 
users  to  produce  a  variety  of  simulations  more  rapidly  without  having  to 
master  a  variety  of  computer  and  mathematics  skills.  To  enhance  its  usability, 
the  program  will  be  compliant  with  the  graphical  user  interface  of  the 
platform  on  which  it  is  running.  The  deliverables  for  this  component  of  the 
project  will  be  the  software  design  authoring  package  for  the  SGI  workstation 
family. 

•  Surgery  simulation  interaction  application 

Taking  the  same  core  simulation  code  used  by  the  authoring  program,  a 
program  is  being  developed  that  uses  the  surgery  simulation  content  created 
by  the  authoring  program  to  allow  a  trainee  to  carry  out  a  surgery  simulation. 
This  program  will  be  a  stand-alone  program  that  does  not  require  special 
computer  skills  to  perform  the  surgery.  The  user  will  work  with  interface 
devices  that  allow  them  to  work  in  the  manner  they  expect,  e.g.  hold  a  scalpel 
not  a  mouse  to  cut  a  tissue  in  the  simulation.  This  application  will  initially  be 
implemented  on  Silicon  Graphics  workstations.  Toward  the  end  of  the 
project  an  attempt  will  be  made  to  implement  this  on  a  more  consumer 
oriented  platform  and  operating  system  such  as  a  Pentium  machine  with 
dedicated  graphics  accelerator  and  Windows  NT.  The  interface  between  the 
user  and  the  computer  will  be  a  general  spatial  tracking  and  force  feedback 
device  that  can  be  used  for  all  the  tasks  of  the  procedure.  The  deliverable  for 
this  component  is  the  interactive  simulation  engine  for  the  SGI  workstation 
family. 

•  Tools  to  create  anatomic  geometry  from  Visible  Human  data  set 

Specific  tools  are  being  added  to  the  software  authoring  toolkit  to  enable  the 
user  to  create  the  geometry  of  the  needed  anatomy  for  the  "Shattered  Kidney" 
procedure  from  the  Visible  Human  data  set  of  the  National  Library  of 
Medicine.  In  addition,  these  tools  will  allow  the  user  to  take  other  data,  such 
as  MR  and  CT  scans,  and  use  this  to  create  the  geometry.  Enhancements  of 
these  tools,  which  will  be  developed  by  HT,  will  be  added  later  in  the  project 
as  they  become  available  from  our  subcontractor,  SRI  International.  The 
deliverables  for  this  component  include: 

•  Initial  manually  controlled  software  tools 

•  Enhanced  tools  incorporating  semi-automated  techniques 
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•  Tools  to  create  realistic  textures  for  anatomy 

Tools  are  being  developed  to  create  photorealistic  texture  maps  for  the 
anatomic  structures  used  in  the  simulation.  These  tools  will  allow  us  to 
create  detailed  surface  features  that  provide  visual  cues  required  to  effectively 
navigate  the  surgical  field  during  the  procedure.  Commercial  3D  paint 
software  will  be  adapted  if  something  with  the  appropriate  features  can  be 
found,  otherwise  we  will  develop  our  own  software  and  use  the  color 
information  contained  in  the  photographic  data  of  the  Visible  Human  data 
set  to  guide  the  creation  of  the  anatomy  coloring.  The  deliverable  for  this 
component  is  a  fully  functional  surface  painting  software  module  for  Teleos. 

•  Integration  of  force  feedback  equipment  with  the  simulation  code 

The  current  force  feedback  technology  has  been  evaluated  and  we  have 
determined  that  the  PHANToM  (SensABLE  Devices,  Cambridge,  MA)  is  the 
most  appropriate  device  for  use  with  the  "Shattered  Kidhey"  procedural 
simulation.  A  cooperative  development  partnership  has  been  established 
with  SensABLE  Devices  to  ensure  that  the  device's  capabilities  and  its 
software  interface  fully  meet  our  simulation  requirements.  A  prototype 
application  has  been  developed  to  test  out  deformable  object  algorithms  with 
a  force  feedback  device.  The  device  has  been  integrated  into  the  simulation 
interaction  application  built  upon  the  core  code  so  that  force  feedback  can  be 
available  in  any  authored  simulation.  The  deliverables  for  this  component  of 
the  project  include: 

•  Prototype  of  demonstration  of  force  feedback  with  a  deformable  object 

•  Full  integration  of  force  feedback  into  Teleos  simulation  interaction 
application 

Surgery  Content  Development  Tasks: 

•  Creation  of  normal  anatomy  for  open  kidney  procedure  simulation 

The  Teleos  authoring  software  and  the  software  developed  by  its 
subcontractor,  SRI  International,  have  been  used  to  create  anatomy  needed 
for  the  simulation  from  the  Visible  Human  data  set  (please  see  page  15  for 
more  details).  This  anatomy  will  be  delivered  in  stages  with  only  the  amount 
being  created  that  is  appropriate  to  the  subsections  of  the  entire  procedure 
being  simulated  at  a  given  time.  The  data  includes  the  geometry  of  the 
anatomy  as  well  as  the  surface  coloring/ texture  information.  The  deliverables 
for  this  component  of  the  project  are  multiple  sets  of  anatomy  delivered  for 
each  subsection  of  the  procedure. 


10 


(2)  Software  Feature  Set 


The  software  design  authoring  environment  has  been  created  for  the  non¬ 
programmer  who  has  some  facility  with  graphics  applications  that  run  on  SGI 
machines.  This  individual  most  likely  works  in  a  medical  school,  university, 
hospital,  medical  research  laboratory  or  commercial  environment  as  a 
medical  content  developer.  The  extensive  use  of  'scripting'  and  object- 
oriented  design  allows  the  developer  to  rapidly  assemble  procedural 
simulations. 

Importing  files  from  other  applications:  In  addition  to  the  native  file  format, 
a  number  of  popular  3D  file  formats  will  be  supported  by  the  software 
authoring  environment,  specifically  Inventor™  (SGI)  and  its  subsets  (e.g., 
VRML).  Inventor^M  contains  a  file  translation  program  to  convert  other  files 
into  the  Inventor  format.  For  example,  Silicon  Graphics  Object  (SGO)’^'^  data 
files  can  be  translated  into  Open  Inventor"^"^  files  using  the  translator. 

Modeling  and  model  attributes:  The  Modeler  program  contains  several 
functions  for  importing  models,  building  models,  texture  mapping,  adding 
multimedia  components  such  as  audio,  timing  functions,  viewing  functions, 
assigning  physiological  properties  and  attaching  physical  properties. 
Deformable  models  are  built  and  manipulated  in  the  context  of  the  Modeler. 
Physical  characteristics  such  as  stiffness,  mass  and  damping  can  be  assigned  to 
the  models.  Other  tools  are  available  for  building  3D  models  from  2D  images 
(accepts  all  popular  2D  image  file  formats).  Each  model  is  assigned  its  own 
attribute  list,  including  collision  parameters,  intrinsic  physics,  texture, 
etcetera. 

The  Blood  Source  Builder  assigns  bleeding  and  blood  attraction  and  repulsion 
attributes.  It  contains  a  flow  rate  slider,  bleeding  delay  control,  color  wheel 
(e.g.,  for  distinguishing  between  arterial  and  venous  blood)  and  cut  shaping 
functions.  Blood  attractor  and  repulsor  points  can  be  placed  at  any  location  in 
the  virtual  environment,  and  have  a  slider  control  for  strength  and  polarity. 
The  Visual  Features  Module  allows  importation  of  2D  texture  maps,  assigns 
built-in  textures  including  bone,  cartilage,  artery,  vein,  muscle  type  (skeletal, 
cardiac,  smooth),  nerve  (CNS,  PNS,  white  matter,  gray  matter,  choroid  plexus, 
meninges),  intestine,  liver,  spleen,  pancreas,  kidney,  lymph  node  and  tissue, 
urinary  tract,  reproductive  tract,  respiratory  tract  (lung,  bronchi),  skin,  fascia, 
connective  tissue  and  adipose  tissue,  among  others.  Additional  texture 
mapping  controls  include  placement  and  specular  effects.  The  ViewWorld 
Module  includes  all  features  necessary  for  viewing  models,  including  scene 
lighting  and  cameras.  The  Timing  Module  includes  event-driven 
phenomena,  such  as  clocks  that  can  triggered  for  each  object  and  event.  Pop¬ 
up  text  dialog  boxes  can  also  be  linked  to  objects  when  events  occur,  for 
example,  this  will  allow  the  designer  to  assign  a  text  "WARNING"  with 
explanation  to  a  mistake  event.  The  Audio  Recorder  Module  allows  the  user 
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to  record  sound  bites  and  assign  them  names  for  use  in  the  World  Sequencer 
(see  below).  All  the  events  are  controlled  and  scheduled  using  the 
Sychronization  Module. 

Building  virtual  environments:  The  GUI  Driven  World  Sequencer  allows  the 
designer  to  import  models  and  establish  relationships  between  models  to 
build  the  completed  virtual  environment.  Connective  elements  such  as 
springs  can  be  used  to  tie  models  together  and  constrain  their  movement.  It 
also  allows  assignment  and  queuing  of  up  to  4  audio  tracks.  Active 
manipulators  can  be  selected  from  7  choices  -  scissors,  retractor,  forceps,  knife 
(includes  scalpel),  probe,  autosuture  and  clip /clamp  applier.  Scripting 
capabilities  are  also  available  for  added  functionality. 

Input  device  integration:  The  Device  Interface  Integrator  allows  the  designer 
to  select  the  interactive  device  to  be  used  when  a  new  environment  is  opened 
up.  Supported  devices  include  mouse  (default),  PHANToM  (SensABLE 
Devices),  Immersion  laparascopic  tools  and  endoscope  (Immersion 
Laboratories)  and  the  Polhemus  Fastrack. 

Other:  Features  include  standard  edit  functions  (cut,  copy,  paste,  save,  save  as, 
etc.),  display  options  (stereo  or  monaural)  and  on-line  help. 

(3)  Software  Architecture 

The  software  authoring  toolkit  is  a  development  environment  for 
developing  production  quality  software  for  interactive  simulation  of  surgery. 
Typical  scenarios  in  surgical  simulation  include  human  organs  -  such  as  skin, 
urinary  tract,  heart,  lungs,  arteries,  veins,  blood  -and  the  results  of  their 
interaction  with  surgical  devices,  such  as  rupturing  of  skin  and  tissues, 
cutting  of  arteries  and  veins  and  blood  flow  as  a  result  of  that,  deformation  of 
the  surface  on  contact  with  the  device.  The  software's  approach  to  this 
challenge  is  through  physically-based  modeling  of  flexible  rigid  bodies. 
Physically-based  modeling  incorporates  physical  characteristics  into  models, 
permitting  mathematical  simulation  of  their  behavior  [Barzel,  1992]. 

The  architecture  is  driven  by  the  potential  packages  and  programs  (Stand 
Alone  Projects  or  SAP's)  that  will  be  developed  on  top  of  the  system. 
Internally  it  is  a  set  of  efficient  and  highly  reusable  C++  classes  built  over 
IRIX,  OpenGL  and  XLib/Xt/IM.  For  ease  of  implementation,  the  authoring 
toolkit  has  been  conceptually  divided  into  two  parts:  HtTypes  Library  and 
Primitives  Library.  Built  on  top  of  these  will  be  the  packages  that  can  be  used 
for  surgery  simulations  by  customers. 

HtTypes  Library  is  a  library  of  classes  which  are  independent  of  OpenGL  and 
X,  and  includes  vectors,  matrices,  etc.  The  Teleos  Primitives  Library  is  built  on 
top  of  HtTypes  Library,  OpenGL,  and  XLib/Xt/IM.  The  software  packages  built 
using  the  software  authoring  toolkit  are  referred  to  as  Stand  Alone  Projects. 
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An  example  of  an  SAP  is  the  "Shattered  Kidney"  simulator  that  is  being 
developed  for  the  present  project. 

B.  "Shattered  Kidney"  Abdominal  Simulation 

A  "Shattered  Kidney"  formally  refers  to  multiple  lacerations  of  the  kidney 
with  contained  or  disrupted  fragments,  often  held  together  by  the  remains  of 
the  renal  capsule  (McAninch  and  Carroll,  1989).  This  usually  results  from 
severe  blunt  or  penetrating  trauma  to  the  kidney.  McAnnich  and  Carroll 
(1989)  provide  the  standard  method  for  classifying  the  extent  of  renal  trauma. 
Among  civilians,  80%  of  all  renal  trauma  is  due  to  blunt  trauma,  with  motor 
vehicle  accidents  accounting  for  the  vast  majority  of  cases  (Chambers, 
Champion  and  Edson  (1989).  Although  the  kidneys  are  well  protected  by  the 
ribs,  vertebrae,  back  muscles  and  abdominal  viscera,  they  are  the  most 
commonly  injured  organs  in  abdominal  trauma.  In  the  battlefield  setting,  the 
"Shattered  Kidney"  can  be  caused  by  extensive  penetrating  tvounds,  such  as 
caused  by  bullets  or  shrapnel,  or  by  massive  blunt  trauma.  In  both  the 
emergency  room  and  in  the  battlefield,  the  best  approach  for  the  "Shattered 
Kidney"  is  rapid  complete  or  partial  nephrectomy  (Guerriero,  W.G.  and  M. 
Coburn  (1994)). 

(1)  Procedure  (adapted  from  Taylor,  D.L.  and  W.R.  Fair  (1985)) 

Major  indicator  of  serious  complications:  Loss  of  continuity  of  the  renal 
capsule  and  distraction  of  fragments. 

If  visualized  or  suspected,  major  perirenal  hematoma  indicates  the  necessity 
for  opening  of  the  retroperitoneum  and  exploration  of  the  kidney.  In  the 
emergency  setting,  injudicious  exploration  may  lead  to  unnecessary 
nephrectomy,  however,  several  studies  have  shown  the  first  2-3  days  offer  the 
best  window  of  opportunity  for  corrective  surgery. 

Surgical  approaches: 

(1)  Make  a  vertical  midline  incision  in  the  abdomen. 

(2)  Begin  exploration  of  the  abdomen,  following  establishment  of  hemostasis 
-  include  debridement  of  the  path  of  any  penetrating  object(s)  or  missile(s). 
The  retroperitoneum  is  approached  along  the  aorta  at  the  root  of  the 
mesentery. 

(3)  If  you  find  the  existence  of  a  large  retroperitoneal  hematoma  or  other 
indications  of  a  major  renal  trauma,  open  the  retroperitoneum  medial  to  the 
inferior  mesenteric  vein  and  identify  the  aorta. 
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Indications  for  surgical  exploration  of  major  trauma  (McAninch  and  Carroll 
(1989)): 

•  Expanding  or  uncontained  hematoma 

•  Pulsatile  retroperitoneal  hematoma 

•  Major  urinary  extravasation 

•  Non-viable  renal  parenchyma  (loss  of  more  than  15%  of  the  kidney) 

•  Vascular  injury 

Please  note  that  some  of  these  indications  can  only  be  determined  using  pre¬ 
operative  imaging  methods,  which  may  not  be  practical  in  a  battlefield 
setting. 

(4)  Dissect  cephalad  along  the  anterior  surface  of  the  aorta  to  expose  the  left 
renal  vein  crossing  the  aorta. 

(5)  Vascular  control  is  obtained  by  clamping  the  renal  vein  and  artery  at  their 
origins  from  the  vena  cava  and  the  aorta,  (mistake  possible  here  -  if  the 
surgeon  strays  too  far  cephalad  and  lateral  to  the  aorta,  the  splenic  artery  and 
vein  may  be  mistaken  for  the  left  renal  artery  and  vein). 

(6)  Reflect  the  colon  prior  to  exploration  of  the  retroperitoneal  space. 

(7)  Use  the  principles  of  exploration  outlined  by  Scott  and  Carlton  and  their 
colleagues  (Scott,  Carlton,  and  Goldman  (1969);  Scott  and  Selzman  (1966): 

•  Obtain  early  vascular  control 

•  Debride  any  devitalized  tissue 

•  Obtain  hemostasis  by  suture  ligation 

•  Insure  watertight  closure  of  the  collecting  system 

•  Approximate  the  parenchymal  margins 

•  Obtain  extraperitoneal  drainage  of  the  renal  fossa 

(8)  Analysis/corrective  surgery  of  the  kidney: 

Substantive  damage  limited  to  a  portion  of  the  kidney  usually  indicates 
partial  nephrectomy. 

Disruption  of  the  collecting  system  usually  occurs  through  damage  to  the 
calyceal  fornices.  This  repair  involves  watertight  suturing  of  the  collecting 
system  at  the  level  of  the  fornix. 
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If  bilateral  renal  damage  is  present,  the  kidney  with  the  best  possibility  of 
recovery  is  repaired  first. 

Transcapsular  damage  is  often  present  in  major  renal  injuries,  especially 
those  with  penetrating  injuries  (eg,  bullets,  shrapnel;  see  Scott,  Carlton  and 
Goldman  (1969)).  In  these  cases: 

•  Ligate  all  major  vessels. 

•  Use  a  watertight  closure  for  the  collecting  system. 

•  Close  the  capsule  with  running  or  interrupted  sutures. 

•  Use  diverting  pyelostomy  for  clots  in  the  renal  pelvis. 

•  Make  sure  that  all  devitalized  tissue  is  debrided. 

(2)  Simulation  issues 

(a)  Overview 

The  "Shattered  Kidney"  procedure  presents  several  technical  challenges  for 
medical  simulation.  First,  it  is  an  open  field  surgical  procedure,  so  that  cutting 
open  of  the  abdominal  wall  will  be  one  of  the  first  steps  in  the  procedure. 
Cutting,  retraction  and  finally  suturing  closure  will  have  to  be  simulated. 
Second,  arteries  and  urinary  tract  components  may  have  to  be  ligated,  which 
involves  a  high  dexterity,  two-handed  motor  manipulation.  Finally,  the 
actual  nephrectomy  involves  the  extensive  cutting  and  removal  of  a  large 
amount  of  tissue,  necessitating  handling  of  large  tissue  masses  within  the 
context  of  the  simulation  (i.e.,  mediated  by  the  input  device(s)). 

A  script  and  storyboards  for  the  simulation  are  under  development.  In 
addition  to  the  technical  challenges  in  physically-based  as  outlined  above  and 
solved  as  part  of  Teleos  development,  interactive  multimedia  elements  are 
being  added  to  serve  as  audio  and  text  cues  for  the  user  of  the  simulation.^ 
These  include  "Warning"  cues  to  notify  the  user  when  mistakes  are  made,  or 
when  a  component  of  the  procedure  is  taking  longer  than  appropriate. 
Finally,  performance  tracking  and  management  software  is  being  developed 
to  aid  in  the  evaluation  of  user  performance.  The  emphasis  is  on  timing  and 
accuracy  of  motor  manipulations,  and  on  cognitive  aspects  of  the  simulation. 
Registration,  administration  and  results  screens  will  allow  the  user,  or  test 
administrator,  to  evaluate  the  performance  of  an  individual  user  and 
compare  his  or  her  performance  to  a  database  of  expert  and  novice  users. 

(b)  Extraction  of  the  Visible  Human  data  for  abdominal  modeling 

One  of  the  tasks  that  has  been  completed  is  the  creation  of  models  of 
abdominal  organs  for  the  "Shattered  Kidney"  procedure.  The  normal 
anatomy  has  been  extracted  from  the  "Visible  Human"  dataset  obtained  from 
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the  National  Library  of  Medicine,  National  Institutes  of  Health.  The  ultimate 
goal  of  the  Visible  Human  Project  was  the  acquisition  of  CT  and  MRI  images, 
and  high  resolution  photographic  images  of  transverse  cryostat  sections,  from 
a  representative  male  and  female  cadaver  at  a  resolution  of  one  millimeter  or 
greater.  The  Visible  Male  data  set  is  approximately  15  GB,  and  was  obtained 
from  the  NLM. 

Dr.  Harlyn  Baker,  Senior  Computer  Scientist  at  SRI  Interanational,  has 
developed  automated  approaches  for  extracting  and  segmenting  the 
abdominal  anatomy  for  the  "Shattered  Kidney"  simulation,  as  part  of  the 
subcontract  component  of  this  grant.  Several  anatomical  reconstructions 
have  been  made  with  the  cross-sectional  photographic  images.  These  images 
are  2048  X  1216  pixels  in  size  in  24  bit  color.  3D  models  are  being  derived  from 
this  data  set  through  the  process  of  polygonalization.  A  mesh  is  created  from 
the  high  resolution  color  data,  and  the  points  define  the  vertices  of  the 
polygons.  Anatomical  regions  that  have  alot  of  detail  are  represented  by  a 
denser  meshwork  of  polygons.  Once  the  polygonal  mesh  has  been  created  for 
a  given  2D  section,  it  is  aligned  over  consecutive  sections  to  create  a  3D 
model.  The  process  of  alignment  (called  registration)  is  not  trivial,  since  even 
a  small  amount  of  mis-registration  may  cause  spatial  breaks  in  small  objects 
such  as  blood  vessels.  Once  the  3D  model  has  been  created,  it  needs  to  have 
other  features  such  as  physical  properties  and  texture  maps  added  to  the 
spatial  framework.  Finally,  the  model  needs  to  be  manipulated  in  such  a  way 
that  it  will  provide  a  realistic  simulation  of  a  surgical  procedure  or 
anatomical  demonstration.  All  of  these  events,  from  3D  model  building 
through  surgical  simulation,  are  achievable  in  the  context  of  the  software 
authoring  toolkit. 
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III.  CONCLUSIONS 


The  following  conclusions  can  be  drawn  from  the  first  year  of  the  project: 

(1)  The  development  of  appropriate  software  tools  is  critical  to  the  success  of 
the  project.  We  have  developed  a  software  authoring  environment  in 
which  it  is  possible  to  author  interactive,  physically-based  simulations 
with  the  physiological  realism  necessary  to  simulate  a  procedure  as 
complicated  as  the  "Shattered  Kidney".  Within  the  context  of  this 
software,  specific  technical  challenges,  including  cutting,  blood  flow, 
suturing,  tissue  deformation,  and  force-feedback  appropriate  for  the 
"Shattered  Kidney"  procedure  are  being  addressed  and  resolved.  Future 
progress  depends  on  a  combination  of  algorithmic  and  graphics 
computing  solutions  to  resolve  demanding  components  of  the  procedure. 

(2)  The  PHANToM  input  device  will  be  used  as  the  appropriate  equipment 
interface  for  the  simulation,  because  it  offers  the  greatest  degree  of  control 
and  sensitivity  available  on  the  market.  It  is  being  programmed  to  provide 
appropriate  levels  of  force  feedback  and  haptic  sensitivity  for  the 
procedure. 

(3)  Appropriate  levels  of  anatomical  detail  for  abdominal  structures  has  been 
achieved  using  the  "Visible  Human"  dataset  which  has  been  obtained 
from  the  National  Library  of  Medicine.  The  photographic  image 
resolution  is  high  enough  for  the  most  detailed  features  of  the  simulation, 
and  the  dataset  can  be  decimated  (reduction  of  complexity)  for  portions  of 
the  simulation  that  require  higher  speed  and  less  detail. 

(4)  The  educational  and  instructional  design  issues  relevant  to  the  "Shattered 
Kidney"  simulation  have  been  identified  and  addressed  in  the  context  of 
the  simulation.  The  most  important  features  to  be  implemented  in  the 
future  are:  (1)  acquisition  of  appropriate  motor  and  cognitive  skills  by  the^ 
user  of  the  simulation,  and  (2)  software  recognition  of  user  error  using 
embedded  performance  measures. 
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